Stereotactic computer assisted surgery based on three-dimensional visualization

ABSTRACT

A computer assisted surgical system that includes an apparatus (110) for imaging a region of interest of a portion of an anatomy of a subject; a memory containing executable instructions; and a processor programmed using the instructions to receive two or more two-dimensional images of the region of interest taken at different angles from the apparatus and process the two or more two-dimensional images to produce three dimensional information associated with the region of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a national phase entry under 35 U.S.C. § 371of International Application No. PCT/EP2009/050210, filed Jan. 9, 2009,published in English, which claims the benefit of U.S. ProvisionalPatent Application No. 61/010,543, filed Jan. 9, 2008. The disclosuresof said applications are incorporated by reference herein.

BACKGROUND OF THE INVENTION

The invention relates to a system and method of computer assistedsurgery (CAS) using stereotactic navigation with three-dimensionalvisualization, and more specifically to a CAS system that is reactiveand does not disrupt operating room workflow procedures.

A current method of inserting implants (consisting of, for example, aplate and associated screws) is typically accomplished by positioningthe plate on the corresponding anatomical location and inserting thescrews with the assistance of fluoroscopy. The implantation of platingand nailing systems is often a difficult task because operating room(OR) procedures are generally minimally invasive and thus placement isachieved by trial and error using fluoroscopy such as a C-arm apparatus,i.e., C-arm vision. This generally leads to long operating times.Furthermore, during such a procedure, both the patient and the operatorare exposed to significant amounts of radiation.

In addition, in some cases it may be impossible to determine theposition of implant components (for example, the screws in the bone)with sufficient precision because the fluoroscopic image is onlytwo-dimensional. This may lead to misplacement or insertion of screws ofimproper length. This, in turn, may cause high revision rates or eveninjuries (e.g., hip joint injury). In order to ensure that these implantcomponents do not extrude from the bone, it is thus sometimes necessaryto position these implant components with an excessively large margin oferror away from the edge of the bone. In many instances, the result isthat the implant cannot be positioned as intended, and the desiredbiomechanical stability cannot be achieved. In the case of femoral neckfractures, for example, the use of conventional fluoro-navigation doesnot result in any significant improvement.

Other cutting edge technologies currently being used in operating roomsto assist in surgery include intra-operative three-dimensional (3D)imaging and navigation systems based on tracking technology. However,only a few hospitals are using these technologies. The limited adoptionof these technologies is primarily due to their high cost, the effortinvolved in installing these systems, and the significant resultingchanges to OR procedures or workflow. For example, tracking technologiesrequire a line of sight between the tracking device and navigationdetection system. This disrupts the normal workflow since the surgeonand other personnel must then remain cognizant of the system's line ofsight requirements.

Further, as a general matter, satisfactory positioning of a mainimplant, like a plate or nail, cannot be defined pre-operatively. Forexample, during an operation positioning can be done by haptic match onthe bone surface or by reaming the bone to make space for anintra-medullary nail. In addition, although the position ofsub-implant(s) might be based only on pre-operative images (e.g.,fluoroscope or CT images), such position is still relative to theposition of the main implant. Thus, a positioning procedure cannot becompletely planned pre-operatively, but must be optimized during theoperation. In this regard, classical stereotaxis cannot be used due tothe fact that the position cannot be predefined.

Accordingly, there is a need for a computer assisted surgery (CAS)system that enhances surgical procedures without significantlydisrupting the normal OR workflow. More specifically, there is a needfor a combined 3D imaging and CAS system which can be easily and readilyintegrated into the clinical environment. Preferably, such a systemwould be low cost, easy to set-up and use, and minimize changes to theOR workflow.

SUMMARY

An aspect of the present invention is a reactive method for stereotacticsurgery. The method preferably comprises positioning an implantassociated with a reference body on a region of interest of a patient'sanatomy; detecting information associated with the implant using animaging system; determining, based on the detected informationassociated with the implant, an action to be taken as part of thesurgery; and displaying positional information associated with theimplant and the region of interest based on the action to be taken.

In accordance with this aspect of the present invention, positioningcomprises acquiring two fluoroscope images of the region of interest attwo different angles.

Further in accordance with this aspect of the present invention,displaying further comprises processing detected information associatedwith the implant by estimating the contours of the region of interest inat least two dimensions based on the plurality of two-dimensionalimages.

Further still in accordance with this aspect of the present invention,detecting comprises detecting the presence of the reference body basedon one or more fiducial markers.

In another aspect, the present invention is a method for stereotacticsurgery. The method preferably comprises positioning a medical deviceassociated with a reference body proximate a region of interest of aportion of an anatomy of a subject and imaging the region of interest attwo or more angles to obtain a plurality of two-dimensional images. In apreferred embodiment, the reference body comprises a plurality offiducial members, most preferably at least four such markers that arevisible to the imaging system. It is further preferred that the fiducialmarkers comprise spheres that are visible to the imaging system.

In accordance with this aspect of the present invention, the pluralityof two-dimensional images are processed to produce three dimensionalinformation associated with the region of interest. In addition, themethod further preferably includes associating, based on the threedimensional information, a virtual medical device with the region ofinterest and the reference body and displaying the association as animage showing the virtual medical device superimposed onto the region ofinterest.

Further in accordance with this aspect of the present invention, thevirtual medical device comprises a main implant and one or moresub-implants. In addition, the virtual main implant is superimposed overthe current location of the actual implant and the virtual sub-implantsare generated so as to show its future position. Accordingly, thevirtual sub-implants inform the surgeon of where the actual sub-implantwill be located before it is placed in the region of interest.

In accordance with this aspect of the present invention, imagingpreferably comprises acquiring two fluoroscope images of the region ofinterest at two different angles. In addition, processing furtherpreferably comprises estimating the contours of the region of interestin at least two dimensions based on the plurality of two-dimensionalimages.

Further in accordance with this aspect of the present invention,processing may further comprise forming a three dimensional imageassociated with the region of interest based on the estimation. In afurther preferred aspect, the present invention may be applied to asurgical implant procedure wherein the region of interest comprises afemoral head, the plurality of two dimensional images compriseanterior-to-posterior and axial images of the femoral region andestimating comprises forming an outline of the femoral head on theanterior-to-posterior and axial images. In this regard, the method mayfurther comprise forming parts of a three dimensional sphererepresenting important portions of the femoral head.

Further still in accordance with this aspect of the present invention,the medical device preferably comprises an intracapsular plate and thereference body is connected to the plate, and positioning comprisespositioning the intracapsular plate on a femur proximate the femoralhead. In addition, the virtual medical device preferably comprises avirtual intracapsular plate and displaying comprises showing the virtualintracapsular plate superimposed on the position of the intracapsularplate in relation to the femoral head.

In another aspect, the present invention is a computer assisted surgicalsystem, comprising: an apparatus for imaging a region of interest of aportion of an anatomy of a subject; a memory containing executableinstructions; and a processor programmed using the instructions toperform a method. In this regard, the processor preferably receives twoor more two-dimensional images of the region of interest taken atdifferent angles from the apparatus, processes the two or moretwo-dimensional images to produce three dimensional informationassociated with the region of interest, superimpose a virtual referencebody onto the region of interest based on the three dimensionalinformation to form an image showing the virtual reference body relativeto the region of interest, and generate a display signal associated withthe superimposed image. Preferably, the reference body is first detectedand superimposed onto an object that models the region of interest,e.g., sphere for a femoral head; and the display signal is thengenerated.

In accordance with this aspect of the present invention, the processorpreferably processes the one or more two-dimensional images by outliningthe contours of the region of interest in two dimensions and creates athree dimensional object representing the region of interest. The threedimensional object may be derived from a database and based on age andgender of the patient. The three dimensional object may also bedetermined based on landmarks associated with the region of interest.

Further in accordance with this aspect of the present invention, amedical device may comprise a device selected from the group consistingof an intracapsular plate, an artificial joint, a pacemaker and a valve.

In another aspect, the present invention is a system and method ofcomputer assisted surgery (CAS) using stereotactic navigation withthree-dimensional visualization, wherein an implant or implant systemacts as a stereotactic device. The invention provides a reactive CASsystem designed for use with mono-axial and poly-axial plates and nails.Based on the principles of stereotactics and 2D-3D matching, a system isprovided that virtually suggests or gives indication of the optimalposition of an implant by calculating such position. In addition, thesystem may also calculate screw lengths before drilling. Aided by imageprocessing and virtual 3D visualisation, the system can achieve optimalbiomechanics.

In addition, unlike existing navigation systems, the CAS system of thepresent invention is designed to be reactive to reduce any additionaleffort for the surgeon. In particular, the system may be triggered byuse of a reference body, implant K-wires, or screws that are normallyused as part of the surgical procedure. In addition, by detecting thesedevices, the system is able to determine the step in the workflow that'sbeing performed. More specifically, image processing is used to detectvarious objects during the workflow and determine which step is beingperformed by the surgeon and for system adaptation.

In another aspect, the system provides necessary 3D information withoutthe need for intra-operative 3D imaging (e.g. 3D C-arms). The system isalso low cost, easy to set-up and use, and minimizes changes to the ORworkflow. The present system also requires fewer X-ray images and istherefore safer for patients.

In another aspect, the invention makes use of an iterative procedure(which for the example of using an ICP to fix a femur neck fracture)includes one or more of the following steps:

-   -   1. positioning an implant in an anatomical region of interest,        e.g., based on a satisfying haptic match;    -   2. fluoroscopic imaging of the anatomical region of interest;    -   3. virtually checking the future position of the sub-implant(s);    -   4. virtually realigning the implant according to constraints        until a satisfactory virtual position is reached;    -   5. providing active or passive realignment values for position        of the implant to the surgeon (i.e., actively by identifying the        best location or passively by allowing the surgeon to decide;    -   6. actual realignment of the plate by the surgeon based on        realignment values and a satisfactory haptic match; and    -   7. iterate procedure starting at step 2 until operation        complete.

These and additional aspects and features of the present invention aredescribed in further detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a stereotactic computer assisted surgical system inaccordance with an aspect of the present invention.

FIG. 1B depicts a computer that may be used in the system of FIG. 1 inaccordance with an aspect of the present invention.

FIG. 2 illustratively depicts how a conventional two-dimensional (2D)image may not accurately show the positions of screws in a region ofinterest.

FIG. 3 illustratively depicts how three-dimensional imagery may be usedto depict the positional information that is not apparent withconventional two dimensional images.

FIG. 4A is a flowchart illustrating a procedure for implanting a medicaldevice in accordance with an aspect of the present invention.

FIG. 4B is a flowchart illustrating a procedure for positioning animplant in accordance with an aspect of the present invention.

FIG. 4C is a flowchart illustrating a procedure for generating a virtualimage of an implant and a region of interest.

FIG. 5 shows the placement of an intracapsular plate implant inaccordance with an aspect of the present invention.

FIG. 6A is a side view of an implant system including a reference bodyand an implant in accordance with an aspect of the present invention.

FIG. 6B is perspective view of a reference body and an implant inaccordance with an aspect of the present invention.

FIG. 6C is perspective view of a reference body and an implant inaccordance with an aspect of the present invention.

FIG. 7 illustrates the placement of an intracapsular plate implant ontoa femur.

FIG. 8 illustratively depicts the step of taking two fluoroshots fromdifferent angles.

FIG. 9 illustrates detection of the femur head in the two fluoroshots.

FIG. 10 illustrates the visualization of a virtual three dimensionalsphere representing the femur head based on conical projections of thetwo dimensional fluoroshots.

FIG. 11 shows the step of displaying a visualization based on a matchingof the three dimensional sphere with the two dimensional images.

FIG. 12A shows a step of automatically adjusting the proposed positionof the intracapsular plate in the distal direction.

FIG. 12B shows a step of automatically adjusting the proposed positionof the intracapsular plate in the distal direction.

FIG. 13A shows a step of automatically adjusting the proposed positionof the intracapsular plate by external rotation.

FIG. 13B shows a step of automatically adjusting the proposed positionof the intracapsular plate by external rotation.

FIG. 14 shows the re-positioning and fixation of the intracapsular platebased on the proposed position.

FIG. 15 shows automatic detection of the insertion of K-wires anddetecting movement of the femur head to ensure a reactive behavior.

FIGS. 16A-16H illustrate use of the method of FIG. 4 in accordance withan aspect of the present invention.

DETAILED DESCRIPTION

Generally, in one aspect, the system of the present invention is basedon the registration of fluoroscopy images with an implant associatedwith a reference body. For example, the implant (e.g., an angle stableplate) may include the reference body or may be positioned in apredefined location in relation to the reference body, which is detectedor recorded in a fluoro image. Thus, the actual spatial dimension andposition of the implant can be determined by means of the correctidentification and registration of the reference body in the fluoroimages.

Where multiple related implants are included as part of the procedure,e.g., main implants and sub-implants, after registration of the mainimplant as described above, the location of any remaining sub-implantsmay be depicted virtually in the correct spatial position in relation tothe fluoro images of the main implant. The sub-implants (e.g., screws ofthe associated angle stable plate) will be located in a fixed,pre-defined position in relation to the main implant after all implantshave been implanted.

In order to provide the information necessary for an anatomicallycorrect location of all (main and sub-) implants, important anatomicalregions are approximated using three dimensional bodies or objectsdepicted in the fluoro image in correct relative position. Target valuesare compared with the values of the location of the remaining implants,which are used in determining the current position of the main implant.

During pre-operative planning (for example, using a non-invasive appliedreference body), the partial or sub-implants (e.g., screws) can first beplaced in an optimum position, independent of the location of the mainimplant (plate). In a subsequent operation (using an invasive referencebody), where the main implant location has been determined by thepre-operative planning (with a position estimate derived by thesurgeon), the location of the main implant can be optimized using hapticfeedback. After the registration as described above, the resultinglocation of the partial or sub-implants are depicted virtually; thisposition is compared to the position of the partial implant in thepre-operative plan, and to the distances to important anatomical(three-dimensional) structures. In a reactive iterative process(adjusting the plate as instructed by the system), it is possible todetermine the optimum balance between an ideal main implant location(for example, plate fit) and the ideal partial implant position (forexample, screw location).

Turning now to FIG. 1A, there is illustrated a stereotactic computerassisted surgical (CAS) system 100 in accordance with an aspect of thepresent invention. As shown, in the preferred embodiment, the system 100includes an imaging apparatus 110, such as a C-arm fluoroscope, and acomputer device 120 such as laptop computer. In general, the computerdevice 120 contains a processor 150, memory 160 and other componentstypically present in general purpose computers as depicted in FIG. 1B.

Memory 160 stores information accessible by processor 150, via bus 162for example, including instructions 164 for execution by the processor150 and data 166 which is retrieved, manipulated or stored by theprocessor 150. The memory 160 may be of any type capable of storinginformation accessible by the processor 150, such as a hard-drive, ROM,RAM, CD-ROM, write-capable, read-only, or the like. The instructions 164may comprise any set of instructions to be executed directly (such asmachine code) or indirectly (such as scripts) by the processor. In thatregard, the terms “instructions,” “steps” and “programs” may be usedinterchangeably herein. The functions, methods and routines of theprogram in accordance with the present invention are explained in moredetail below.

Data 166 may be retrieved, stored or modified by processor 150 inaccordance with the instructions 164. The data may be stored as acollection of data. For instance, although the invention is not limitedby any particular data structure, the data may be stored in computerregisters, in a relational database as a table having a plurality ofdifferent fields and records, or as an XML document. The data may alsobe formatted in any computer readable format such as, but not limitedto, binary values, ASCII or EBCDIC (Extended Binary-Coded DecimalInterchange Code). Moreover, any information sufficient to identify therelevant data may be stored along with the data, such as descriptivetext, proprietary codes, pointers, or information which is used by afunction to calculate the relevant data.

Although the processor 150 and memory 160 are functionally illustratedin FIG. 1B within the same block, it will be understood by those ofordinary skill in the art that the processor 150 and memory 160 mayactually comprise multiple processors and memories that may or may notbe stored within the same physical housing. For example, some or all ofthe instructions 164 and data 166 may be stored on removable CD-ROM andothers within a read-only computer chip. In addition, some or all of theinstructions 164 and data 166 may be stored in a location physicallyremote from, yet still accessible by, the processor 150. Similarly, theprocessor 150 may actually comprise a collection of processors which mayor may not operate in parallel.

As shown, computer device 120 may comprise additional componentstypically found in a computer system such as a display (e.g., an LCDscreen), user input (e.g., a keyboard, mouse, game pad, touch-sensitivescreen), microphone, modem (e.g., telephone or cable modem), and all ofthe components used for connecting these elements to one another.

As is also shown in FIG. 1A, a patient 185 would typically be positionedon an operating table with various restraints such that the area to beoperated on is constrained from moving during the surgery. Thefluoroscope 110 (or other suitable imaging apparatus) is used to obtainimages of the region of interest of the patient's anatomy, e.g., theregion being operated on or area to which the implant will be attached.As is discussed in further detail below, an illustrative region ofinterest may comprise an area that includes the femoral neck and anintracapsular plate (ICP). The computer 120 (or other suitable imageprocessing and display apparatus) is used to process the images from thefluoroscope, determine positioning for the implant and sub-implants, andprovide feedback/instructions to the surgeon. The processing stepsperformed by the computer are described below.

In another aspect, the present invention addresses a problem with thecurrent technique of ICP implantation of accurately positioning theplate using two-dimensional (2D) images. This problem is in part due tothe dangerous screw placement needed to avoid cutouts. Specifically, theends/tips of the screws need to be set as close as possible to thesecond cortex. However, the 2D images used by the surgeon do not reflectthe 3D nature of the problem.

FIG. 2 shows common drawbacks of a conventional two-dimensional (2D)image. In particular, a 2D image 200 may not indicate improperpositioning of a screw. In this instance, the 2D image 200 makes itappear that the screws are correctly positioned within the bone.However, a 3D illustration could provide additional information showingthat a screw may actually have perforated the bone. For example, FIG. 3illustrates how 3D imagery may expose a problem with screw positioningthat is not apparent with conventional 2D imaging. In FIG. 3, neither ofthe 2D images 300, 310 shows a problem with the screw position. However,if the 2D images are combined to create a 3D visualization, it becomesapparent the tip of the screw projects through the bone as illustratedat 320. Hence, the 2D imagery currently relied upon by a surgeon may notalways accurately reflect the location and positioning of medicaldevices and the like within a region of interest. Thus, in this example,it would be beneficial for a surgeon to have access to 3D imagery.

In one aspect, the present invention provides a system and method whichgenerates 3D information from the 2D imagery to allow for more accuratepositioning of a medical device, e.g., an implant, and thereby avoidingthe above problems. Generally, as used herein, the term medical deviceincludes any biomedical device or structure that is introduced orimplanted into the anatomy of a subject. Such devices include those thatreplace or act as missing biological structures, or that are placed overor within bones or portions of the anatomy. As mentioned above, thepresent invention is described using the illustrative example ofimplanting an intracapsular plate (ICP) to repair a femoral neckfracture. Note, however, that the invention may find application innumerous surgeries, including virtually all fields of bone surgery(e.g., trauma, orthopedics, and pediatrics).

By way of background, it is generally known that fractures are usuallyrepaired by reduction and fixation of the broken bones. The individualfragments of bone are aligned in their normal anatomical position (i.e.,reduced) so that separated parts can grow together again. It isnecessary that the parts remain relatively stable with respect to eachother over an extended period of time to allow for healing. In somecases, particularly for more complicated fractures, it is necessary toconnect the individual broken bone pieces directly to one another. Inthese cases, the fracture is fixed or reduced via an invasive procedurewherein an implant is installed within the body with screws or nails.

Turning now to FIG. 4A, there is depicted a high level flowchart 400 ofthe method steps of implanting an implant in accordance with an aspectof the present invention. As shown, the method begins with positioningof a main implant in a region of interest, at step S402. As is explainedin further detail below, this initial positioning is preferably doneusing fluoroshots taken along at least two dimensions or directions.Once the main implant is positioned to the surgeon's satisfaction, thesystem 100 generates an image showing the position of a virtual implantand associated virtual sub-implants relative to the region of interest,step S408, based on the fluoroshots and the position of a reference bodyor reference objects within the field of view of the fluoroscope 110.

Using the image of the virtual implants, the surgeon may then affix theimplant, using the sub-implants for example, as is depicted at S424.Once the sub-implants (e.g., screws) and implants are in the place, thesystem may perform a quality check, at S428, by detecting and displayingthe actual location of these implants relative to their desiredposition. This quality check is desirable given that duringimplantation, the position of an implant or sub-implant may change fromits ideal position due to the mechanical forces during, for example,drilling or screw placement or as a result of movement by the patient.In this regard, quality checks, such as step at S428, may also beperformed during affixation of implant, at step S424. Additionally,quality checks may also be performed post operatively using the systemto detect movement in the implant caused by, for example, patientactivity.

Significantly, the above method 400 is reactive in that the surgeon isnot required to inform the system 100 of which step he/she is performingas part of the OR workflow. In this regard, this system 100 iscompatible with the normal OR workflow and is able to determine the stepin the OR workflow that is being performed by, for example, detectingthe presence of a reference body or object.

Turning now to FIG. 4B, there is depicted the sub-steps or procedure forpositioning or aligning the implant in accordance with step S402 of FIG.4A. As shown, the procedure begins with insertion and positioning themain implant in the anatomical region of interest at step S430. Inkeeping with the illustrative example, an intracapsular plate (ICP) isused to repair fractures of the femoral neck. Thus, at step S430, theICP would be inserted into the patient and roughly positioned on thebone, in this case the femoral neck. This step may be done, for example,in accordance with the normal OR workflow, such as by allowing thedoctor to use haptic feedback to judge an appropriate initial positionfor the plate.

In this regard, FIG. 5 illustrates the placement of an ICP implant 510(e.g., the main implant) along with sub-implants (i.e., the screws) tosecure a femoral neck fracture. As shown, the ICP 510 is affixed to thefemur 520 and screws are inserted through the neck and head of the ICP.Preferably, in accordance with an aspect of the present invention, thescrews entering the femoral head are positioned entirely within thehead. In an additional aspect of the present invention, as the ICP iscontoured to the shape of the femur, the degrees of freedom inpositioning the ICP on the bone are limited and are used as part of thealignment procedure S402. Specifically, the ICP can only be shiftedalong (i.e., translation) and/or rotated around the shaft axis of thefemur. In addition, the ICP has threaded holes so that theposition/angle of the screws relative to the plate is known.

In accordance with an aspect of the present invention, prior toinsertion of the main implant within the region of interest, the mainimplant 510 is connected to a reference body or object. The referencebody is preferably attached to (or part of) the implant, but may also beattached to an aiming device or instrument (e.g., a drill guide). Inthis way, the position of the implant may be determined based on thelocation and position of the reference body. Preferably, each implant isassociated with a different reference body that is detectable by thesystem 100, in particular the fluoroscope 110. In a preferredembodiment, the reference body comprises a plurality of sphericalfiducial markers inserted on or in the instrument (e.g., aiming device).By arranging the fiducial markers in a predetermined pattern, they mayserve as identifiers for different instruments. In this regard, thefiducial markers and instrument may be conveniently referred to as areference body—though the fiducial markers are what provide thereference.

For example, FIG. 6A illustrates a side view of a reference body 604 aspart of the implant 610. Together, the reference body and implant arereferred to herein as an implant system 614. As shown in FIG. 6A, thereference body 604 includes a one or more fiducial markers 616 that aredetected by the imaging system and used as points of reference ormeasurement. Preferably, the fiducial markers comprise spheres to makefor easier detection in a two-dimensional imaging system such as afluoroscope. In addition, the arrangement of the fiducial markers withinthe reference body functions as signature that is used to identify thereference body and the associated implant. Given that the dimensions ofthe reference body and implant are known and these devices are fixedrelative to each other, the location of the implant can be accuratelydetermined by detecting or recording the location of the reference body.As is also shown in FIG. 6A, fiducial markers may also be placed on theimplant itself, but are not necessary.

FIG. 6B is a perspective view of the aiming device 604 and implant 610(which in keeping with the example is an ICP) in a detached condition.FIG. 6C shows these two devices in an attached condition. As is shown,the aiming device 604 is contoured to fit the ICP 610. In addition, itincludes openings that allow access to the screw holes on the ICP 610that are used to secure the implant 610 as is explained in furtherdetail below. To allow for processing, the reference body must be in thefield of view of the image with the implant and the region of interest640, which in keeping with this illustrative example includes the femurneck and femoral head. As part of this initial insertion and placement,the surgeon will typically use haptic feedback to determine a startinglocation for the implant.

FIG. 7 shows an instrument 700 that may be used in the insertion andinitial placement of the ICP 610 onto a femur 720. In addition, inmaking this initial placement, the surgeon may use either instrument 700or reference body 604. In this regard, the instrument 700 may alsocomprise a reference body by placing the appropriate fiducial markers onor in it.

Returning to FIG. 4B, once the surgeon determines an initial locationfor the implant (and accompanying reference body), a fluoroshot is takenof a region of interest along a first dimension or direction, at step434. For example, a fluoroshot may be taken along the anterior-posteriordimension or the axial dimension. With reference to FIG. 1, theanterior-posterior view is illustrated with the source 190 and detector192 aligned along the y-axis, while in the axial view the source anddetector are aligned along the z-axis As shown in FIG. 8, fluoroshotsmay be taken from any two different dimensions or directions 810A, 810B.Preferably, the two images will be taken perpendicular to one another(i.e. close to a 90 degree angle between them), but this is not requiredand any angle will suffice.

As the implant and reference body are located within the field of viewof the imaging device, given their proximity to region of the interest,the fluoroscope 110 detects the presence of the reference body, i.e.,the fiducial markers. Computer 120 then uses the image data it receivesfrom the fluoroscope 110 to provide a visualization of the location ofthe implant relative to the region of interest. In particular,registration of the fluoroscopic images is performed using the referencebody. As discussed above, the reference body is typically in a fixedposition relative to the implant and bone. In addition, a disk is usedat the image intensifier to develop the X-ray image. The disk is usuallydirectly attached to the image intensifier. Such a reference body wouldalso be seen in every image and could be used to compensate fordistortion of the image and for determination of the center of, forexample, the X-ray beam.

Determination of the implant relative to anatomical region of interestis done using known image processing techniques based on the variationin the spatial radiation arriving at the detector, including theradiation directed at the region of interest and reference body. Usingthe spatial variation, the computer is able to construction an imagethat accurately depicts the spatial relationship between the implant andregion of interest (e.g., femur and femoral head) as a two dimensionalimage.

Upon viewing this image, the surgeon may then determine if the implantshould be re-positioned, as at step S438. For example, the surgeon maydecide to adjust the position along the length of the femur closer tothe femoral head or other degree of freedom. If the surgeon decides suchan adjustment is warranted, he/she repositions the implant as is shownat step S440 and additional fluoroshots are taken at step S434. On theother hand, if the surgeon determines that the no adjustment is neededalong in this dimension the procedure continues at step S442 withstabilization of the implant. In keeping with the example, stabilizationcould be effected by insertion of a Kirshner wire (K-wire) through oneor more openings in the ICP.

With the implant fixed as described above, a fluoroshot may then betaken along a different dimension, step S446. In particular, if thefluoroshots in step S434 were taken along the anterior posteriordirection, in step S446 they may be taken along the axial direction orat another angle. In this regard, as part of step S402, it may besufficient to use a single image for this step to optimize positionalong only one degree of freedom (e.g., a distal shift of the implant)where 3D information is not needed.

Upon completion of the fluoroshot at step S446, the surgeon may thenview an image of the position of the implant. If it is determined thatthe implant needs to be adjusted at step S448, e.g., rotated in the caseof an ICP, the procedure returns to step S446 and additional fluoroshotsare taken along this dimension. Once the surgeon is satisfied that theimplant is appropriately positioned based on images obtain along thisdimension, the procedure continues at step S450 with additionalstabilization of the implant. For example, where the implant or medicaldevice is an ICP, K-wires may be inserted through additional openings inthe ICP. As result of the foregoing procedure, the position of the ICPor other implant may be positioned by the surgeon iteratively and inaccordance with normal OR workflow procedures. That is, the surgeon mayrepeat any steps within the procedure until the implant is appropriatelypositioned.

With the implant positioned as described above in relation to step S402,the method then continues as shown at step S408 of FIG. 4A and as willbe now described in further detail as shown in FIG. 4C. In particular,at step S454, the system may then generate 3D information from the twodimensional fluoroshots recorded in step S402 or additional twodimensional fluoroshots may be taken at different angles as describedabove. Since the implant is now stabilized relative to the region ofinterest, additional fluoroshots may be taken with the K-wires acting asa trigger for the system. In addition, as the reference body may also beattached to the implant, it may also serve as a reference object asdescribed above.

In accordance with this aspect of the present invention, the resulting2D images are processed to locate and outline a three dimensionalcontour, i.e., a sphere, of the femoral head. For example, FIG. 9 showsan AP view image 910 and an axial view image 920 with superimposedcircles 930, 940 outlining the contours of the femoral head. The circles930, 940 may be constructed by computer 120 using image processingtechniques such as edge detection or computer generated models. Suchmodels may be created pre-operatively using MRI or other non-invasivetechniques that can determine the location and size of organs or boneswithin the region of interest.

In addition, using the 2D images, the computer 120 then determines andgenerates 3D object that is associated with and models the region ofinterest, step S456, in accordance with another aspect of the presentinvention. In particular, FIG. 10 illustrates the visualization of avirtual 3D sphere representing the femur head based on conicalprojections of the 2D images 910, 920. As shown in FIG. 10, the virtual3D sphere is formed by projecting the two dimensional coordinate systemonto a three dimensional coordinate system. In this example, as theoutline of the femoral head forms circle, the projection onto athree-dimensional coordinate system results in a sphere. Depending onthe contours of the region of interest, these projections may be doneusing a Cartesian and/or spherical coordinate system. In addition, thelocation of the object in relation to region of interest may beaccurately determined based on the position of the implant in relationto the reference body.

FIG. 11 shows the step of displaying a visualization of the region ofinterest, implants and sub-implants based on a matching of the 3D spherewith the 2D images, step S458. As shown in FIG. 11A, the inventionsuperimposes a virtual ICP 1104 with screws and a spherical outline ofthe femoral head onto the original 2D axial and AP images. Thisvisualization allows the surgeon to readily see the position of the ICPand screws relative to the femoral head. Notably, the visualizationshows the positions of virtual screws, their length and how they will bepositioned within the femoral head. In addition, the system can suggestthe screw, e.g., particular model, or screw length that would besuitable for affixing the implant.

The visualization also allows the surgeon to manually adjust theposition of the actual ICP if better alignment is considered necessary.For example, FIG. 12A shows a proposed adjustment of the ICP 510 in thedistal direction, as seen from the AP two dimensional image. Morespecifically, as can be also seen from FIG. 12B, the display may alsoinclude zones that indicate preferable translational adjustment of theimplant relative to its current location. For example, an acceptancezone 1220 (e.g., using colors) may be used to indicate a more preferablelocation. Thus, if the screws are located outside this area distally,the surgeon may manually adjust the plate at S55 and view an updatedvisualization by returning to step S52.

FIGS. 13A and 13B show how adjustment of the ICP 610 by externalrotation 1300 can be achieved. In particular, similar to the case oftranslational adjustment, if the surgeon believes that the implant isnot aligned properly, he/she may adjust arrows 1300 to visualize how theimplant is aligned if rotated. As can be also seen from FIG. 13B, anacceptance zone 1340 may be used to show how rotation would change thelocation of the screws from a preferred position. As is also shown inFIG. 13B, rotational movement is preferably performed after the locationalong the femur neck has been satisfactorily determined. In this way,the reference body 604 may fixed to the bone using a pin 1326. This pin1326 may then be used with teethed gear mechanism 1330 to more preciselyrotate the implant as shown.

As discussed above, the present invention is reactive in that the systemreacts to the surgeon in lieu of requiring the surgeon to take action orinteract with the computer or system. As such, if the surgeon decidesthat the implant is properly aligned, he/she can then decide to securethe implant and complete the procedure. This minimizes disruptions incurrent OR workflow and allows the surgeon to use his/her judgment aspart of the workflow. In contrast, conventional approaches tend todisrupt the OR workflow by requiring the surgeon to interact with theCAS. This typically lengthens the surgical procedure and requires morein the way of equipment, both of which increase the cost of surgicalprocedures.

Upon completion of the steps outlined above in relation to step S408,the procedure continues to step S424, where the implant may be affixedto the region of interest. Additional fluoroshots may be taken during orafter step S424 to verify reduction of the fracture and the position ofthe ICP and K-wires or screws. For example, FIG. 15 shows the insertionof K-wires 1620, 1630 and a detected movement of the femur head 1610.Ideally, the system will detect such movement and propose a correctiveaction such as new screw lengths. If necessary, further re-positioningof the implant and reduction of the fracture can be performed asdiscussed above.

As discussed above, Kirshner wires (K-wires) can be inserted throughopenings in the reference body 604. More specifically, as shown in FIG.15, a first K-wire (1630 may be inserted to fix the plate to the bone.(A second K-wire 1620 may also be inserted through the fracture.) Screwscan then be inserted to compress the fracture (S424). The screws may beself-tapping or be inserted through drilled holes. The inventionpreferably includes image processing software that is based on edgedetection to detect any insertion and bending of the screws or K-wires.Such software may comprise a component or routine in a set ofinstructions that carry out the method described above. The ICP hasthreaded screw holes so that the screw position/angle relative to theplate is fixed. The K-wires may be removed before or after the screwshave been inserted.

Alternatively, an aiming apparatus with scaling in combination with anoblong-shaped hole (wherein a K-Wire may be inserted) may be directlyattached to the ICP and used to assist in the mounting and any furtheradjustment deemed necessary by the surgeon. FIG. 14 shows repositioningusing a prior art one-shot apparatus 1400; which is preferably replacedby the reference body 604 and other attachments 1326, 1330 discussedabove.

Turning now to FIGS. 16A-16H, there is shown an alternative use of themethods described above in accordance with a further aspect of thepresent invention. As will be discussed in detail, these figures showinsertion of a locking nail 1654, which is used as part of a hipfracture repair system. In particular, and with reference FIG. 16A, theprocedure begins with insertion of the nail 1654 into the femur 1658. Asis also shown, the nail 1654 is attached to an instrument or aimingdevice 1660. The aiming device is preferably equipped with a pluralityof fiducial markers, e.g., four or more, that act as a reference bodythat is detectable by the imaging system. In accordance with the methodspreviously described, at this initial state of the procedure, thesurgeon obtains fluoroshots along a first dimension. For example, afluoroshot may be obtained along the anterior-posterior axis of thepatient or at any other angle the surgeon deems suitable.

As is shown in FIG. 16B, if the shot is taken along theanterior-posterior direction, the computer 120 detects calculates theposition of the reference body and instrument 1660, and displays avirtual nail 1666 in relation to the region of interest 1670. Inaddition, the display includes a projection 1674 of the location of ascrew that will be used to secure the nail 1654 within the femoral heador region of interest 1670. As is also depicted in FIG. 16B, if theprojected path 1674 of the screw is determined by the surgeon to requiresome adjustment, the surgeon may make a translational adjustment 1678 ofthe nail within the femur 1658. After making the translationaladjustment 1678, the surgeon then, preferably, takes an additionalfluoroshot to confirm that the adjustment moved the nail 1654 into amore desirable position using they display 1680 similar to that shown.

Once the surgeon is satisfied with translational alignment of the nail1654, he may then use the system to rotationally align the nail as isshown in FIG. 16C. In particular, the surgeon would take a fluoroshot ata different angle, such as along a hip lateral direction to obtain theimage 1684 shown in FIG. 16C. Using this image, the surgeon may rotatethe nail 1654 into a more desirable position and take additionalfluoroshots to confirm the adjustment.

Once the surgeon determines that the nail 1654 is suitably aligned, hemay then insert a K-wire 1687 as is shown in FIG. 16D. With the K-wireinserted, two or more two-dimensional images may be obtained with thefluoroscope, as described above. Using these two or more images, thesystem is then able to determine the appropriate screw lengths, as isshown in FIG. 16E. In particular, the two-dimensional images are used tocreate an object that models the region of interest, in this case, thefemoral head. More specifically, where the region of interest is thefemoral head, the computer 120 uses these two-dimensional images tocreate a sphere 1689 and superimposes within the sphere the location andlengths of screws that may be used to attach the nail 1654. As is shownin FIG. 16E, the display includes a virtual screw 1691 along with tickmarks 1693 that indicate the length of the screw just within the sphere1689 and out through an opening in the nail 1654.

Based on the tick marks 1693 shown in FIG. 16E, the surgeon may thenselect an appropriate screw of desirable length to affix the nail 1654.Once the screw is selected, it is then inserted as is shown in FIGS. 16Fand 16G. As is also described above, once a screw is in place,additional images may be taken to verify that the length of the screwsecure the device without protruding outside the region of interest as aresult of the forces that were applied during the affixation procedure,as is illustrated in FIG. 16H.

The image processing performed by the invention includes: anatomicfeature detection and segmentation; position detection of the referencebody; generation of 3D information from 2D images; registration,rendering and display of 3D information on 2D images; and calculation ofthe optimal position of the implant. In addition, in another aspect, thesystem may propose an appropriate length for each screw.

As discussed above, at least two 2D images containing the reference bodyare required by the invention to provide 3D information. These imagesshould be taken at different angles (preferably near a 90 degree angle).Additional 2D images can also be used to provide information. The imagescan be registered to one another by detecting distinctive anatomicfeatures in the images and/or by using the reference body. The referencebody (which occurs in each image) can be used to precisely register theimages in three dimensions. The reference body can also be helpful inautomatically detecting these anatomic structures for segmentation (e.g.detecting feature borders). The relative position of specific anatomicalstructures to the position of the reference body may also be estimatedbased on general bone shape statistics and on patient data (e.g. size,gender, age). This relative position may be used as a starting point forthe segmentation algorithms. Once the anatomic structures have beensegmented, the image processing software can correlate the structuresfrom different images to generate 3D information.

Various three-dimensional reconstruction algorithms can be used togenerate this information. Typically, the algorithms will approximatethe segmented anatomic features with geometric shapes (e.g., a circle).The geometric shapes are then matched/registered to their known relativepositions in the 2D images. These shapes are then projected into 3Dspace to form, for example, a sphere or cylinder. The invention mayinitially select a typical 3D Shape for an anatomic region from adatabase and match it with the image by zooming, rotating, and/ortranslating the shape. The shape may also be altered, such as with amorphing algorithm, for a better match. In fact, pre-operative imagesmay be taken of the same anatomic region to better determine the actualshape of various features.

Because the reference body is located within each image and is attachedto an anatomic region (e.g. a bone), movement of the patient duringsurgery is not a problem in accordance with an aspect of the presentinvention. This is because the system can use the location of thereference body to register different fluoroscope images (independent ofthe image content) and generate a low artifact real 3D image using 3Dreconstruction algorithms. This aspect of the invention to preciselyregister the images significantly reduces artifacts due to patientmovement during surgery.

Preoperative planning may be performed by taking pre-operative imagessimilar to the intra-operative images. This pre-operative planning canbe used to determine the optimal sub-implant positioning which may thenbe checked against the intra-operative positioning. Such pre-operativeimages could be processed using different algorithms which are too timeconsuming to use during surgery or could be segmented and matchedmanually.

As discussed above, the invention may also provide a reactive workflowby automatically detecting the status of an operation and thus knowingthe next operative steps to be performed. In this manner, the inventionmight provide suggestions to the surgeon. For example, the invention maysuggest a specific type, size, or shape of a best-fit implant based onthe detected geometry of a fracture. Moreover, the invention couldmodify a previous suggestion based on additional information determinedduring the surgery.

Additional distinctive aspects of the invention include that thestereo-tactic device is implanted in the body. In addition, theinvention uses 2D images (e.g. fluoroscopic x-rays) to generate 3Dinformation. The reference plate (ICP) is contoured to match the surfacecontour of the bone to restrict the degrees of freedom for adjustments.The reference plate (ICP) is also threaded so relative screw position isknown. The invention calculates and proposes reference plate position,sphere position, screw position and lengths.

Advantages of the invention include that it reduces the surgery time forinsertion of an implant, requires almost no interaction between thesurgeon and the system, provides three-dimensional information onimportant regions, requires little change to operating room procedures,and is cheaper than current tracking based navigation. Additionalfeatures of the invention include that it takes into account any bendingof Kirshner wires (K-wires) through automatic detection, calculates anddisplays any dislocation of the femur head during implantation, andcalculates the screw lengths.

Although the invention herein has been described with reference to anICP procedure, it is to be understood that this embodiment is merelyillustrative of the principles and applications of the presentinvention. It is therefore to be understood that numerous modificationsmay be made to the illustrative embodiment and that other arrangementsmay be devised without departing from the spirit and scope of thepresent invention.

The invention claimed is:
 1. A method for operating a computer assistedsurgical system for implanting a medical device having a main implantand a sub-implant, and being represented by a virtual medical device,the method comprising: implanting the main implant coupled to areference body having at least four fiducial markers arranged in apredetermined three-dimensional pattern, the reference body being in aknown spatial relationship with the main implant, the fiducial markersdetectable by a fluoroscopic imaging system; fluoroscopically imaging aregion of interest and the main implant associated with the referencebody to produce first and second two-dimensional fluoroscopic images,the first and second fluoroscopic images representing the position ofthe main implant proximate the region of interest of a portion of ananatomy of a subject at a first angle to obtain the firsttwo-dimensional image and at a second angle to obtain the second twodimensional image to produce three dimensional information associatedwith the region of interest; detecting the presence of the referencebody having the at least four fiducial markers in the first and secondimages; repositioning the main implant and reference body; processingone additional two-dimensional fluoroscopic image with respect to therepositioned reference body fiducial markers by identifying andregistering the fiducial markers of the reference body in the oneadditional two-dimensional fluoroscopic image, to produce locationinformation of the main implant superimposed over the region ofinterest, wherein the location information of the main implant isdetermined based solely on the location of the fiducial markers in theone additional fluoroscopic image; associating, based on the locationinformation of the main implant superimposed over the region ofinterest, the not yet implanted sub-implant with the region of interestin a predetermined relationship relative to the reference bodyrepresenting the main implant, the main implant and the sub-implanthaving a fixed predefined spatial relationship to one another uponimplantation; and displaying the association of the main implant, thenot yet implanted sub-implant and the region of interest as an imageshowing the virtual medical device superimposed over the region ofinterest such that the not yet implanted sub-implant is displayed ashaving a virtual fixed predefined spatial relationship to the mainimplant that is the same as the fixed predefined spatial relationshipbetween the main implant and the sub-implant upon implantation.
 2. Themethod of claim 1, wherein the reference body is attachable to one ofthe main implant and a targeting device, wherein the reference body hasa characteristic two-dimensional projection.
 3. The method of claim 1,further comprising: associating, based on the three dimensionalinformation of the main implant superimposed over the region ofinterest, the not yet implanted sub-implant with the region of interestin a predetermined relationship relative to the reference bodyrepresenting the main implant; and displaying the association of themain implant, the not yet implanted sub-implant and the region ofinterest as an image showing the virtual medical device superimposedover the region of interest.
 4. The method of claim 3, wherein theregion of interest comprises a femoral head, the two dimensional imagescomprise anterior-to-posterior and axial images of the femoral regionand estimating comprises forming an outline of the femoral head on theanterior-to-posterior and axial images.
 5. The method of claim 1,further comprising determining a length of the not yet implantedsub-implant for selecting the required sub-implant.
 6. The method ofclaim 1, further comprising detecting the presence of the reference bodybased on the fiducial markers.
 7. The method of claim 1, whereinprocessing further comprises estimating the contours of the region ofinterest in two dimensions based on the one two-dimensional image. 8.The method of claim 7, wherein processing further comprises forming athree dimensional image associated with the region of interest based onthe estimation.
 9. The method of claim 1, further comprising adjustingthe position of the main implant and repeating the steps of imaging,processing, associating and displaying.
 10. A computer assisted surgicalsystem, comprising: an apparatus for imaging a region of interest of aportion of an anatomy of a subject and a medical device having animplanted main implant and a sub-implant being represented by a virtualmedical device; a memory containing executable instructions; and aprocessor programmed using the instructions to: mount a reference bodyhaving a three-dimensional pattern of at least four radiopaque markerstherein onto the main implant in a known relationship thereto; obtain aplurality of two-dimensional images of the region of interest with themain implant associated with the reference body, the image representingthe position of the main implant proximate the region of interest of aportion of an anatomy of a subject at a first and second angle; processthe two-dimensional images to produce three dimensional information ofthe region of interest and the reference body with respect to thereference body radiopaque markers to produce information of the mainimplant superimposed over the region of interest, wherein theinformation includes the actual spatial position of the main implant andthe reference body radiopaque markers and registration of the referencebody in the images; associate, based on the position of the main implantsuperimposed over the region of interest and an image of the referencebody and radiopaque markers, the not yet implanted sub-implant with theregion of interest in a predetermined relationship relative to thereference body representing the main implant, the main implant and thesub-implant having a fixed predefined spatial relationship to oneanother upon implantation; and generate a display signal associated withthe main implant, the not yet implanted sub-implant and the region ofinterest as an image showing the virtual medical device superimposedover the region of interest such that the not yet implanted sub-implantis displayed as having a virtual fixed predefined spatial relationshipto the main implant that is the same as the fixed predefined spatialrelationship between the main implant and the sub-implant uponimplantation.
 11. The computer assisted surgical system of claim 10,wherein the apparatus comprises a fluoroscope.
 12. The computer assistedsurgical system of claim 10, wherein the associated sub-implant providesan estimate of the location of the main implant and one or moresub-implants within the region of interest.
 13. The computer assistedsurgical system of claim 10, wherein the processor processes the one ormore two-dimensional images by outlining the contours of the region ofinterest in two dimensions and creates a three dimensional objectrepresenting the region of interest.
 14. The computer assisted surgicalsystem of claim 13, wherein the three dimensional object comprises asphere.
 15. The computer assisted surgical system of claim 13, whereinthe three dimensional object is derived from a database and based on ageand gender of the patient.
 16. The computer assisted surgical system ofclaim 13, wherein the three dimensional object is determined based onlandmarks associated with the region of interest.
 17. The computerassisted surgical system of claim 10, wherein the system is adapted totake into account bending of Kirschner wires through automaticdetection.
 18. The computer assisted surgical system of claim 10,wherein the system is adapted for calculating and displaying adislocation of a femur head during implantation.
 19. The computerassisted surgical system of claim 10, wherein the system is adapted forcalculating required sub-implant lengths.
 20. The computer assistedsurgical system of claim 10, wherein the system is adapted forautomatically detecting a status of an operation and determining nextoperative steps to be performed so as to provide a reactive system. 21.The computer assisted surgical system of claim 10, wherein the system isadapted to provide suggestions to the surgeon with respect to at leastone out of a group, the group consisting of a specific type, a specificsize, and a shape of a best-fit implant based on a detected geometry ofa fracture.
 22. The computer assisted surgical system of claim 21,wherein the system is adapted to modify a previous suggestion based onadditional information determined during the surgery.
 23. A method foroperating a computer assisted surgical system for implanting a medicaldevice having a main implant and a sub-implant, and being represented ona monitor by a virtual main implant and a virtual sub-implant, themethod comprising: imaging a region of interest with an implanted actualmain implant associated with a reference body in a known spatialrelationship, the reference body including at least four radiopaquefiducial markers mounted on two spaced planes, the imaging representingthe position of the actual main implant proximate the region of interestof a portion of an anatomy of a subject at a first angle and a secondangle to obtain first and second two-dimensional images; processing asingle additional two-dimensional image with respect to the referencebody by identifying and registering the reference body in thetwo-dimensional image using the fiducial markers to produce threedimensional information of the reference and main virtual implantsuperimposed over the region of interest, wherein the information of thevirtual main implant include an actual position of the actual implant;associating, based on the information of the virtual main implantsuperimposed over the region of interest, the virtual sub-implant to beimplanted with the region of interest in a predetermined relationshiprelative to the reference body representing the actual main implant, theactual main implant and the actual sub-implant having a fixed predefinedspatial relationship to one another upon implantation; and displayingthe association of the virtual main implant, the virtual sub-implant tobe implanted and the region of interest as an image showing the virtualmain implant and virtual sub-implant superimposed over the region ofinterest such that the sub-implant is displayed as having a virtualfixed predefined spatial relationship to the main implant that is thesame as the fixed predefined spatial relationship between the mainimplant and the sub-implant upon implantation, and virtually realigningthe main implant and sub-implant until a satisfactory position isreached including the location of the reference body fiducial markersand thereafter implanting the actual sub-implant in the satisfactoryposition based on the location of the fiducial markers in additionalimages.
 24. The method of claim 23, wherein the reference body isattachable to one of the actual main implant and a targeting device,wherein the fiducial markers of the reference body have a characteristictwo-dimensional projection.